Joint transmission of multiple multimedia streams

ABSTRACT

A system and method for multimedia data transmission including pooling, by a computer, at least first and second multimedia data streams together in a single channel, wherein each stream defines a respective sequence of frames and each frame includes at least a base layer and at least one enhancement layer; dynamically establishing, by the computer, first and second bit rates respectively associated with the first and second streams; and transmitting, by a transmitter, the at least first and second streams on the channel.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present U.S. application for Patent is a continuation of U.S. patentapplication. Ser. No. 10/282,797, entitled “JOINT TRANSMISSION OFMULTIPLE MULTIMEDIA STREAMS,” filed Oct. 28, 2002, and assigned to theassignee hereof and hereby expressly incorporated by reference herein inits entirety.

FIELD OF THE INVENTION

The present disclosure relates generally to multimedia transmission.

BACKGROUND

Multimedia such as video and audio can be transmitted over a number ofpaths, including cable, the Internet, and broadcast. For instance,satellite or terrestrial broadcast stations can be used to transmitmultimedia to mobile computing devices such as mobile telephones.

Typically, multimedia data is voluminous, which means that significanttransmission path bandwidth, unfortunately a finite resource, must beused. This is particularly the case for high fidelity multimedia, e.g.,high resolution video. That is, the higher the quality of service beingprovided, the more bandwidth must be used.

As recognized by the present disclosure, several multimedia streams canbe pooled together in a single channel. The channel might have aconstant overall bandwidth in terms of bit rate, i.e., the number ofbits that can be transmitted in the channel per unit time cannot exceedthe “bandwidth” of the channel. Typically, each stream in the channelwill be accorded a fixed fraction of the bandwidth. Accordingly, the bitrate for each multimedia stream typically is fixed.

As further recognized by the present disclosure, however, a multimediastream might require a higher bit rate for a given quality of service(QOS) some of the time and a lower bit rate for the same QOS at othertimes. As an example, in a video stream, when the video sequence makes ahard cut (i.e., a large amount of pixel changes are taking place), ahigher bit rate is required to achieve the same QOS as could be achievedat other times with a lower bit rate when relatively little action istaking place. Or, in terms of audio streams, a higher bit rate might berequired at the beginning of a speech or music segment than at othertimes during the stream to achieve the same QOS.

Having made the above critical recognitions, the present disclosurefurther understands that one multimedia stream in a channel mightrequire a relatively low bit rate at the same time that another streamin the channel might require a relatively high bit rate, and that itconsequently would be desirable to allocate the limited overallbandwidth of the channel in a way that can account for this.

SUMMARY

The present invention is a system and method for partitioning abandwidth of a single channel among plural multimedia streams in a timevarying manner.

Accordingly, a method for multimedia data transmission includes poolingat least first and second multimedia data streams together in a singlechannel. The method also includes dynamically establishing first andsecond bit rates respectively associated with the first and secondstreams, in light of channel bandwidth constraints. The channel may be abroadcast channel. For the purposes of the non-limiting embodimentbelow, a “base layer” may be defined as the most important part of thebits stream which, if successfully received, decoded, and presented tothe user, would result in a baseline level of video, audio, or othermultimedia stream acceptable to the user. On the other hand, an“enhancement layer” would, when combined with the base layer, enhance orimprove the quality, frequency, signal-to-noise ratio, etc. of themultimedia stream when presented to the user, compared to that of thebase layer alone.

In one non-limiting embodiment, each stream defines a respectivesequence of frames and each frame includes a base layer and at least oneenhancement layer. At least one of the enhancement layers is dynamicallyselected for omission from the channel for at least some frames. Theselection can be based on a quality of service (QOS) metric such assignal-to-noise ratio, or by program stream importance. In anillustrative non-limiting example, the enhancement layer selected foromission in a frame is the layer providing the highest overall QOS toits respective stream among the enhancement layers of the other streamsthat are being simultaneously transmitted in that frame. This limits thetotal bandwidth of the combined frames so that it does not exceed thechannel bandwidth, while providing a minimum QOS to each stream.

In another non-limiting embodiment, the streams can be quantizedjointly. In this circumstance, during quantization, a benefit of addingbits to the stream for each stage of quantization is evaluated, and bitsare added (or not) to the stream based thereon, in light of channelbandwidth constraints.

As another alternative, the bit rates can be established to establishsurplus channel bandwidth that is useful for sending ancillary data.

In another aspect, a computer executes logic that includes partitioninga bandwidth of a single channel among plural multimedia streams in atime varying manner.

In still another aspect, a system for transmitting at least first andsecond multimedia streams in a single channel includes means forestablishing a first bit rate for the first stream during a first frameof the first stream. The system also includes means for establishing asecond bit rate for the first stream during a second frame of the firststream, with the first and second bit rates being different from eachother.

The details of the present disclosure, both as to its structure andoperation, can best be understood in reference to the accompanyingdrawings, in which like reference numerals refer to like parts, and inwhich:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the present architecture;

FIG. 2 is a flow chart of the logic for dynamically establishing the bitrates of the streams during joint quantization of the streams;

FIG. 3 is a flow chart of the logic for dynamically establishing the bitrates of the streams when enhancement layers are used; and

FIG. 4 is a table schematically showing base layers and enhancementlayers of two streams.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring initially to FIG. 1, a system is shown, generally designated10, that dynamically establishes the bit rate of each of pluralmultimedia streams 12 that are combined by a stream combiner 14 andbroadcast together in a single channel by a transmitter 16. Moreparticularly, the bit rates of the streams 12 are dynamicallyestablished in accordance with the logic below by a bit rateestablishing computer 18, prior to being combined into a single channeland transmitted in accordance with principles known in the art.

The non-limiting preferred embodiment shown in FIG. 1 shows a multimediatransmitter 16 that uses wireless means, and more particularly that usescode division multiple access (CDMA) principles. The streams can bebroadcast to plural receivers if desired, or transmitted usingpoint-to-point wireless transmission principles. It is to be understoodthat the present principles apply to other forms of wirelesscommunication such as GSM, TDMA, wideband CDMA, OFDM, etc. as well astransmission of multimedia over cable systems, the Internet, etc. Asused herein in the singular, “multimedia stream” means a single streamrepresenting a single program, e.g., a single music piece or a singletelevision show or movie potentially with accompanying text, images,etc.

As disclosed below, the dynamic allocation of bandwidth can depend onperceived levels of quality based on some objective measure. Quality ofservice (QOS) is one such measure. QOS can be measured in several ways,such as, for example, by signal to noise ratio as expressed in decibels.

Referring to FIG. 2, as indicated at block 20 the streams 12 may bequantized jointly with each other using, e.g., MPEG audio and videoquantization principles known in the art. Under these circumstances thelogic moves to block 22 which indicates that at each stage ofquantization, e.g., for each frame to be quantized, blocks 24 and 26 areundertaken.

Moving to block 24, the benefit of adding more bits to each stream forthe stage under test than are currently allocated are evaluated. Thisevaluation can be done iteratively, starting with a default number ofbits that represents a minimum QOS, for a predetermined number ofiterations, or until a predetermined “benefit” threshold is reached, andalways under the constraint that the total number of bits per unit timeallocated to the plural streams remains within the channel bandwidth. Asmentioned above, the evaluation can be based on a perceived level ofquality based on some objective measure such as QOS as indicated by, forexample, signal to noise ratios as expressed in decibels.

Once the evaluations for the stage under test are completed for eachstream, the logic moves to block 26. At block 26, bits are added to thestream or streams that show the highest improvement in benefit from theaddition of bits, within the constraint of the channel's bandwidth.Stated differently, the bit rates of the streams are dynamicallyoptimized for each frame using a user-defined protocol.

“Quantization” is discussed below. It should be noted that quantizationcould be fixed but may also be variable in addition to the number oflayers used, and that in some embodiments bandwidth allocationconsequently can be fully independent of quantization.

Some schemes such as MPEG4 allow for so-called “enhancement layers” ofbits during quantization. As mentioned above, a “base layer” is providedfor each frame for each stream that represents a minimum amount of datanecessary to render the frame, with enhancement layers for the framebeing available to add fidelity to the image or sound provided by thebase layer. The benefit provided by the increased bit rate resultingfrom the addition of the enhancement layers can be indicated along withthe enhancement layers themselves.

Recognizing this, block 28 of FIG. 3 shows that the base layer of eachframe of each stream, along with the associated enhancement layer orlayers, are received at block 28. Also, a QOS measure is provided alongwith each layer that indicates the amount of “benefit” associated withthe layer. At block 30, to conserve overall channel bandwidth theenhancement layer(s) that add the least benefit for the particular setof stream frames (for “n” streams, there will be “n” frames in the set)are discarded. Other metrics of bandwidth allocation can be used, suchas the relative importance of the competing streams, number ofsubscribers for competing streams, presence of commercials in thestreams, and number of users in a particular zone of coverage who desirea particular stream. Indeed, in some circumstances it may be desirableto remove a particular stream altogether, i.e., base layer andenhancement layer.

FIG. 4 illustrates one preferred, non-limiting logic for undertakingthis. For illustration purposes, FIG. 4 assumes that two streams are tobe transmitted in a single channel, with each stream having a base layerfor each frame and potentially two enhancements layers per frame. Thenumbers in the table of FIG. 4 represent QOS and specifically cumulativesignal-to-noise (SNR) ratios as represented in decibels.

FIG. 4 also assumes that the dynamic bit rate protocol being usedessentially seeks to establish a minimum QOS for each stream for eachframe, and that the channel has a bandwidth that is sufficient only tocarry five of the six layers at a time, i.e., FIG. 4 assumes that thechannel can carry only five layers per transmission period. Accordingly,the enhancement layer providing the highest SNR for a period is omittedin the exemplary non-limiting protocol represented by FIG. 4. That is,the second enhancement layer of the first stream provides a cumulativedb level of “17” for periods 1, 2, 4, and 5, which is higher than thecumulative db level provided by any other layer for those periods, andso is discarded for periods 1, 2, 4, and 5. On the other hand, thesecond enhancement layer of the second stream provides a cumulative dblevel of “16” for period 3, which is higher than the cumulative db levelprovided by any other layer for period 3, and so is discarded for thethird period.

It is to be understood that other protocols and other measures of QOScan be used without departing from the scope of the present disclosure.For example, any enhancement layer providing a cumulative SNR for anyparticular frame or period that is higher than a predetermined thresholdcan be omitted.

Or, as another example, if stream A has a base layer quality of four (4)db and, with an enhancement layer, a quality of eight (8) db, and streamB has a base layer quality of ten (10) db and, with an enhancementlayer, a quality of thirty (30) db, one of two heuristics can be appliedto determine which enhancement layer to omit. If the enhancementproviding the greatest overall benefit of any enhancement layer is to bemaintained, the enhancement layer of stream A would be omitted. However,if it is desired to provide the best quality for the worst stream, theenhancement layer for stream B can be omitted.

Alternatively to allocating the entire channel bandwidth to themultimedia streams, the principles above can be used to reduce thebandwidth needed by each stream so that surplus channel bandwidth isobtained. The surplus bandwidth can be used to send real-time ancillarydata to augment one or more of the streams (e.g., character overlays,graphics, etc.) or to send non-real time ancillary data such as apicture display during a song or ordering information, to augment astream. Yet again, the surplus bandwidth can be used to send otherbroadcast data, system control data, programming data, encryption keys,subscription or programming information, etc.

While the particular JOINT TRANSMISSION OF MULTIPLE MULTIMEDIA STREAMSas herein shown and described in detail is fully capable of attainingthe above-described objects of the present disclosure, it is to beunderstood that it is the presently preferred embodiment of the presentdisclosure and is thus representative of the subject matter which isbroadly contemplated by the present disclosure, that the scope of thepresent invention fully encompasses other embodiments which may becomeobvious to those skilled in the art, and that the scope of the presentinvention is accordingly to be limited by nothing other than theappended claims, in which reference to an element in the singular is notintended to mean “one and only one” unless explicitly so stated, butrather “one or more”. All structural and functional equivalents to theelements of the above-described preferred embodiment that are known orlater come to be known to those of ordinary skill in the art areexpressly incorporated herein by reference and are intended to beencompassed by the present claims. Moreover, it is not necessary for adevice or method to address each and every problem sought to be solvedby the present invention, for it to be encompassed by the presentclaims. Furthermore, no element, component, or method step in thepresent disclosure is intended to be dedicated to the public regardlessof whether the element, component, or method step is explicitly recitedin the claims. No claim element herein is to be construed under theprovisions of 35 U.S.C. '112, sixth paragraph, unless the element isexpressly recited using the phrase “means for” or, in the case of amethod claim, the element is recited as a “step” instead of an “act”.

1. A method for multimedia data transmission, comprising: pooling, by acomputer, at least first and second multimedia data streams together ina single channel, wherein each stream defines a respective sequence offrames and each frame includes at least a base layer and at least oneenhancement layer; dynamically establishing, by the computer, first andsecond bit rates respectively associated with the first and secondstreams; and transmitting, by a transmitter, the at least first andsecond streams on the channel.
 2. The method of claim 1, wherein thechannel is one of: a broadcast channel, a multicast channel, and aunicast channel.
 3. The method of claim 1, wherein the at least one QOSmetric is a signal-to-noise ratio.
 4. The method of claim 1, whereineither stream is constituted using an MPEG principle.
 5. The method ofclaim 1, wherein the streams are quantized jointly.
 6. The method ofclaim 1, further comprising, by the computer, a benefit of adding bitsto the stream for each stream during quantization.
 7. The method ofclaim 1, wherein dynamically establishing the first and second bit ratesis based at least partially on the evaluating act.
 8. The method ofclaim 1, wherein the bit rates are established to provide surpluschannel bandwidth useful for sending ancillary data.
 9. A non-transitorycomputer readable storage medium comprising computer-executableinstructions for execution on a computer to perform a method comprising:pooling at least first and second multimedia data streams together in asingle channel, wherein each stream defines a respective sequence offrames and each frame includes at least a base layer and at least oneenhancement layer; dynamically establishing first and second bit ratesrespectively associated with the first and second streams; andtransmitting the at least first and second streams on the channel. 10.The non-transitory computer readable storage medium of claim 9, whereinthe channel is at least one of: a broadcast channel, a multicastchannel, and a unicast channel.
 11. The non-transitory computer readablestorage medium of claim 9, wherein either stream is constituted usingMPEG principles.
 12. The non-transitory computer readable storage mediumof claim 9, wherein the streams are quantized jointly, and, duringquantization, evaluating for each stream a benefit of adding bits to thestream.
 13. The non-transitory computer readable storage medium of claim9, wherein dynamically establishing first and second bit rates at leastin part depends on the evaluating of the benefit of adding bits to eachstream.
 14. The non-transitory computer readable storage medium of claim9, wherein the bit rates are established to provide surplus channelbandwidth useful for sending ancillary data.
 15. The non-transitorycomputer readable storage medium of claim 9, wherein the method uses atleast one relative importance metric.
 16. A system for transmitting atleast first and second multimedia streams in a single channel,comprising: means for establishing a first bit rate for the first streamduring a first frame of the first stream; means for establishing asecond bit rate for the first stream during a second frame of the firststream, the first and second bit rates being different from each other;means for pooling the at least first and second multimedia data streamstogether in a single channel; means for dynamically establishing firstand second bit rates respectively associated with the first and secondstreams, wherein each stream defines a respective sequence of frames andeach frame includes at least a base layer and at least one enhancementlayer; and means for transmitting the at least first and second streamson the channel.
 17. The system of claim 16, wherein the metric is asignal-to-noise ratio.
 18. The system of claim 16, wherein the streamsare quantized jointly, and further comprising: means for, duringquantization, evaluating a benefit of adding bits to each stream. 19.The system of claim 16, wherein the bit rates are established to providesurplus channel bandwidth useful for sending ancillary data.
 20. Asystem for multimedia data transmission, comprising: a computerconfigured to pool at least first and second multimedia data streamstogether in a single channel, wherein each stream defines a respectivesequence of frames and each frame includes at least a base layer and atleast one enhancement layer, and dynamically establish first and secondbit rates respectively associated with the first and second streams; anda transmitter configured to transmit the at least first and secondstreams.
 21. The system of claim 20, wherein either stream isconstituted using an MPEG principle.
 22. The system of claim 20, furthercomprising a quantizer configured to quantize the streams jointly. 23.The system of claim 20, wherein the quantizer is further configured toevaluate benefit of adding bits to each stream.
 24. The system of claim20, wherein the bit rates are established to provide surplus channelbandwidth useful for sending ancillary data.